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Abstract. Despite a significant improvement in the educational aids 
in terms of effective teaching-learning process, most of the educational 
content available to the students is less than optimal in the context of 
being up-to-date, exhaustive and easy-to-understand. There is a need to 
iteratively improve the educational material based on the feedback col¬ 
lected from the students’ learning experience. This can be achieved by 
observing the students’ interactions with the content, and then having 
the authors modify it based on this feedback. Hence, we aim to facili¬ 
tate and promote communication between the communities of authors, 
instructors and students in order to gradually improve the educational 
material. Such a system will also help in students’ learning process by 
encouraging student-to-student teaching. Underpinning these objectives, 
we provide the framework of a platform named Crowdsourced Annotation 
System (CAS) where the people from these communities can collaborate 
and benefit from each other. We use the concept of in-context annota¬ 
tions, through which, the students can add their comments about the 
given text while learning it. An experiment was conducted on 60 stu¬ 
dents who try to learn an article of a textbook by annotating it for four 
days. According to the result of the experiment, most of the students 
were highly satisfied with the use of CAS. They stated that the system 
is extremely useful for learning and they would like to use it for learning 
other concepts in future. 


1 Introduction 

The emergence of Web 2.0 and Web 3.0 technologies has facilitated development 
of various online environments equipped with the features of social learning[T] [2] [3] 
Due to the ubiquity of the internet, various resources such as electronic text¬ 
books, online forums, MOOCs etc. are accessible at ones fingertips for interactive 
and easy learning [3] . Also, the students have become more comfortable reading 
material online. Although the online teaching/learning method has successfully 
staggered the enrolment of students into such courses and portals, the level of 


participation on the part of students is not satisfactory [5]. The possible reasons 
may include inadequate preparation of online educational systems in terms of 
quality of content as well as the interface [5]. Unfortunately, the education system 
has no provision of making use of the feedback from interaction of students dur¬ 
ing the learning process. This leads to the creation of textbooks which are not in 
sync with the actual requirements of students for active learning!?]. There is an 
immense requirement to eliminate the communication gap between the students, 
instructors and the authors by providing them a common platform, where the 
people from these diverse communities can benefit from each other. 

We aim to provide the framework of an environment, which the authors can use 
to reconsider parts of their textbook modules needing improvement or further 
explanation, after observing the students interactions when they try to under¬ 
stand some part of their textbook. This type of environment will not only help 
the authors in getting constructive feedback about their own content, but will 
also help the instructors to provide a virtual classroom type of environment 
with the features of discussions, brainstorming and feedback to the students. 
Also, they can closely observe the interactions taking place among the students 
and can intervene whenever required. It will help the students in learning various 
concepts with the help of other students which will enable student-to-student 
teaching. Hence, such a portal will be beneficial for three types of stakeholders: 
Authors, Students and Instructors. 

The use of discussion forums is the most prevalent method of collecting knowl¬ 
edge from the people in a crowd. Also, people are known to learn better when 
they participate in discussion forums based websites |5|[5|. Such websites also 
improve the engagement level of the participants!^]. However, there is one con¬ 
straint which is associated with the portals based on discussion forums. Due to 
the non-linear structure of such forums, the user sometimes gets deviated from 
whatever information he was originally searching forjllj. An in-place annotation 
tool can outperform traditional forums as a medium for discussion of classroom 
material. It helps the students to remain in the flow instead of losing context 
on a different forum !T^ |13|. Hence, we suggest the use of annotations in the 
document margins, where the collaboration will happen. 

Keeping in view all these objectives, we propose the framework of a portal named 
Crowdsourced Annotation System (CAS), which provides a facility to the au¬ 
thors to add textbook modules to the content repository, a facility to the instruc¬ 
tors to choose a module from the repository and authorize a group of students, 
and a facility of annotating any part of the text to the students. Chhabra et 
al.!T4] verify the presence of variance of expertise on portals such as CAS, which 
creates an ecosystem among the categories of annotators. The authors demon¬ 
strate that it is this ecosystem, that acts as an impetus in the knowledge building 
process. 

In order to verify the efficacy of CAS, we conducted an experiment with a group 
of students who annotated a textbook module while learning it through the por¬ 
tal. Later, a feedback analysis for CAS was also done, in which, most of the 


students agreed that the portal helped them a great deal in understanding the 
given textbook article. 

2 Related Work 

There has been a significant amount of interest to bring in ICT technology in 
education and learning in general. The contributions in this line are increasingly 
getting interdisciplinary, with academic communities ranging from educational 
psychologists [15] to computer scientists [13] getting involved, thus making this 
domain of research transdisciplinary. 


Education and crowdsourcing Corneli and Mikroyannidis[T6] investigated 
the concept of crowdsourcing in education through an analysis of case studies 
on two open online learning communities- Peer 2 Peer University and Planet- 
Math.org. The authors analyzed various roles played by the individuals involved 
and also envisioned future roles that could be created at these organizations. 
They found that crowdsourcing model provides all the roles which are found in 
the traditional environments, as well as offers some additional richness due to the 
social networking tools. Weld et al. m investigated the potential of crowdsourc¬ 
ing methods for improving online personalized education. The authors discussed 
various challenge areas that offer promise of rapid progress and substantial im¬ 
pact. Porcello and Hsi|18] discussed how multiple user communities and online 
platforms might be coordinated to provide effective experiences in online learning 
environments. 


Annotation Systems In order to eliminate the problems that discussion fo¬ 
rums pose, a few systems have been built by various researchers including CAMILE\19\. 
WebAnn^^, NB System^^ and PAMS 2.0^T\ etc. CAMILE is based on a spe¬ 
cific type of computer-mediated anchored discussion forum, where the threads 
are linked to each other, but it still requires the users to go to another context. 
Although WebAnn provides the advantages of in-context annotations, yet it was 
used just as a record-keeping tool due to discomfort of users in reading online 
material. NB System helped in alleviating the problems faced by WebAnn. It 
is an online portal where users can read and annotate any part of a pdf docu¬ 
ment. PAMS 2.0 is another annotation system based on Web 2.0 technology. In 
their work, the authors demonstrated that the effect of annotation on learning 
achievements becomes more substantial with the use of sharing technique. 


Textbook Creation In order to address the problem of high cost of text¬ 
books for Higher Education in Latin America, Ocha, Sprock and Silveira[52] 
worked on the creation of collaborative open text-books that could be freely 
and legally copied, printed, modified and distributed to students. Baraniuk et 
al[23) developed a portal named Connexions for customized textbook creation. 
However, our work is significantly different from theirs in the sense that they 


provide various textbooks in the form of modules. The students and instructors 
can develop customized textbooks as per their requirements by combining these 
modules. However, we make use of the annotations, which the students add to 
the given part of the textbook in order to understand it. The kind of interaction 
taking place among the students gives an idea to the authors about the parts to 
be reconsidered. Also, this kind of platform helps in the learning process of the 
students. 

3 Crowdsourced Annotation Systeni(CAS)-A Proposal 

We propose a technique of textbook creation and refinement coupled with the 
advantages of annotation systems. By providing an integrated platform in which 
existing content is combined with an interactive environment where the people 
in the crowd collaborate, the quality of material can be immensely improved on 
a gradual ongoing basis. Moreover, using the power of crowd into the content 
generation process will improve the execution speed along with the generation 
of diverse ideas, thereby, improving the quality of content. The motive is to al¬ 
low authors to reconsider the portions of their text to be modihed based on the 
feedback from the crowd. Therefore, in a way, everybody in the crowd becomes 
an author of the text in consideration. Also, it will successfully utilize the in¬ 
telligence of the great masses of the people, which will be a mixture of people 
possessing various types of expertise. 

Essentially, the proposed framework has three major interdependent compo¬ 
nents: (a) a repository of content added by the authors, (b) an interface to add 
in-context annotations to the content, and (c) a mechanism for the collaboration 
of annotators. Essentially, the system is an integration of three major notions: 
crowdsourcing, online textbook assessment and collaborative learning. All the 
stakeholders of the system collaborate on the creation of knowledge in an ex¬ 
tremely dynamic environment. Figure [l] shows the raw textbook posted by the 
author being annotated by the students. The instructors closely observe the an¬ 
notations posted, and intervene to help the students, whenever required. The 
textbook after having been annotated, provides enormous amount of informa¬ 
tion to the author so as to make further modifications to the text as per the 
students expectations, and can easily come out with a modified version. 

We created a web-based annotation system named as Crowdsourced Annota¬ 
tion System (CAS) which depicts the proposed framework. In CAS, a group of 
students are authorised to access a particular article of a particular textbook 
by the instructor. The students communicate with each other by adding anno¬ 
tations. The annotations added, are attached to the corresponding text of the 
article. The text for which there are annotations, is displayed highlighted and 
clicking on this highlighted part displays all the annotations for this text. Also, 
the annotations are displayed on the same page in the document margins, which 
every other student can view and comment. The students can add four types of 
annotations, namely. Questions (A), Answers (A), Insights (I) and Pointers (P). 
Question and Answer type of annotations provide the advantages of a discussion 



Fig. 1. Schematic View of the Textbook Refinement Process. Red, Blue and Green 
lines indicate the purpose for which authors, students and instructors respectively will 
make use of the system. 


forum, however, alleviating the problems of a forum by displaying the discus¬ 
sion in the same context. The annotators can add any extra information about 
some text through the use of Insight annotations. Pointer type of annotations 
provide a facility to refer to some outside relevant material regarding the text. 
These four type of annotations help the students in properly communicating 
their comments about the article being studied. A snapshot of the CAS is shown 
in Figure 



Fig. 2. A snapshot of CAS. (The Crowdsourced Annotation System (CAS) can be 
accessed at http://115.248.248.12/CrowdSourced/) 






























4 The Experiment 


In order to understand how students’ annotations emerge over time, and how 
these can be used in the textbook refinement process, we conducted an exper- 
imenlj^ on a group of 60 CS students from the Indian Institute of Technology 
Ropar, India. The students were authorized to annotate an article on “Ran¬ 
domized Closest Pair Problem” from a reference book[21] for their course on 
Algorithm Design. This article was new to them and was not easy for them to 
understand by independent study. The aim for the students was to understand 
this article with the help of each other. They annotated the article for a dura¬ 
tion of four days. All the transactions happening among the students were stored 
in a database for analysis. Table shows various items of statistics regarding 
the experiment, including the number of annotations, percentage distribution, 
maximum, minimum, mean and standard deviation for the four categories of 
annotations. 


Annotation 

No. Of 

%ae« 

Maximum 

Minimum 

Mean 

Standard 

Type 

Annotations 

Distribution 




Deviation 

1 

444 

2436 

23 

0 

7 

5.9$ 

Q 

339 

1936 

31 

0 

6 

5.86 

A 

953 

5236 

71 

0 

16 

12.83 

P 

92 

536 

7 

0 

2 

1.70 


Table 1. Number, Distribution, Max, Min, Mean and SD of Annotation Types (Our 
analysis data has been kept open at the CAS portal.) 


Also, we recorded certain other parameters of the students’ interaction: 

— There were a total of 15932 reading entries i.e. number of annotations read 
by the students. 

— There was also a feature of up voting or down voting of the annotations. 
There were a total of 811 voting entries. 

— A student could click on a watch button attached to every annotation that 
he added, so as to be informed about the kind of responses that he got in 
future for his own annotations. There were a total of 66 watcher entries. 

— A session denotes a collection of interactions taking place within a particular 
time period. There were 875 sessions with an average duration of 15.49. 

— There were a total of 3159 pageviews which represents the total number of 
pages accessed during the period of 4 days. 


In|14|. Chhabra et al. verify the existence of an ecosystem among the annotators’ 
categories using this experimental data. 



5 User Assessment of CAS 


Emerging information technology cannot be effective if it is not accepted and 
used by potential users. Therefore, at the end of the session, the students were 
asked to provide their feedback about CAS through a web-based form. The form 
used Technology Acceptance Model (TAM) to explore the ‘Perceived Usefulness 
(PU)’ and ‘Perceived Ease of Use(PEOU)’ of CAS[25]. TAM is one of the most 
successful models to assess the user acceptance of emerging information tech¬ 
nology among practitioners and academicians. The model attempts not only for 
prediction but also for the explanation to help researchers and practitioners iden¬ 
tify why a particular system may be unacceptable and pursue appropriate steps. 
PU tells prospective users subjective probability that using a specific application 
system will increase his or her job performance within an organizational context. 
PEOU tells the degree to which the prospective user expects the target system 
to be free of effort. 

The universal efficacy of CAS was also established in the feedback from the par¬ 
ticipants. They reported that the introduction of various types of annotations 
(i.e. I, Q, A, P) in CAS was useful in sharing the knowledge in the group. Dis¬ 
playing the annotations on the same page as the text was useful in bringing 
the students attention towards annotations added by others, which further en¬ 
hanced their knowledge. Apart from these two dimensions, i.e., PU and PEOU, 
the questions were asked on one more dimension named ‘Satisfaction (SAT)\ 
to measure the degree of contentment among the participants. The Eeedback 
Form included fifteen questions using a five-point Likert scale (strongly agree; 
agree; undecided; disagree; strongly disagree). The questions were classified as: 
(1) PU (7 Questions); (2) PEOU (5 Questions); (3) SAT (3 Questions). Out of 
64 participants, 52 had filled the form. On a scale assigning the values as: 5 for 
Strongly agree, 4 for Agree, 3 for Undecided, 2 for Disagree and 1 for Strongly 
Disagree, the participants reported on a mean in the range of 3.12 and 4.25 for 
all the fifteen questions. 

Following are the 15 questions asked in the assessment: 

1. I think the introduction of various types of annotations (i.e. 1, Q, A, P) in 
CAS was useful in sharing the knowledge in the group. 

2. I think with the use of CAS, I was able to understand the given text in less 
time. 

3. I think displaying the annotations on the same page as the text was useful 
in bringing attention towards annotations added by others. 

4. I think CAS was helpful in exchanging my thoughts across others, while 
reading. 

5. I think I have been able to understand the text in a better way through the 
use of CAS, as compared to what 1 would have done, without the use of such 
an interface. 

6. I think that the voting of annotations was useful in scrutinizing high quality 
annotations out of all the annotations. 

7. I think such an annotation system is more useful than a simple discussion 
forum. 


8. I think it was easy to add a new annotation to the text. 

9. I think it was easy to view the annotations added by others in the right 
panel. 

10. I think it was easy for me to understand how to use CAS. 

11. I think the Graphical Interface was quite easy to work with 

12. I think due to the introduction of various categories of annotations, it was 
easy for me to put across my point on the interface. 

13. I am thoroughly satisfied with use of CAS for reading an article. 

14. I would like to use this interface for reading other texts also. 

15. I would like to use this interface in future. 

Figure]^ shows the distribution of various responses of the 15 questions by all 
the 52 students. On an average, 82.69 percent students gave ‘Strongly Agree’ or 
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Fig. 3. Distribution of Responses 


‘Agree’ as their answers. 

We also analysed the suggestions/comments that the students were asked to fill, 
along with the ratings. Following are some of the comments: 

— The idea of having such a system is great, and CAS was good. 

— CAS is very good source to understand texts in a better way. 

— It was user friendly and very useful for understanding while reading the 
article. 

— I would like to test this on further topics. 

— Overall it was a good learning platform where you can easily share your 
understandings with other people in the group, ask questions when required 
and get the answers and point of view of other people. It is a good tool to 
assess and evaluate your understanding of the concept. 

— CAS is very good source to understand texts in a better way. 

— The idea of having such a system is great, and CAS was good. 

— Through this, learning was very easy and quick and also enjoyable. 

— Everything is really good..be it the user-interface or the whole concept of IT... 
Inf act., (sic) I with some of my intellectual friends seriously want to study 
through this medium where exchange of ideas is so easy. 


















— The CAS system was good enough to learn /analyse the text in less time. 

— The site was perfectly built and was very useful to fully understand the topic. 

6 Conclusion 

In this work, we presented the framework of an integrated platform, CAS, which 
can be used by the authors to collect feedback about their textbooks. The text¬ 
books which are created for the students, can be best judged by the students 
only when they try to read them and provide their comments. Also, it provides 
the students a virtual learning environment to help each other understand the 
content. For this purpose, we developed and executed an experiment involving 
text annotations by students on CAS. An online feedback of CAS was later con¬ 
ducted to evaluate how students come to accept and use the system. As per the 
feedback, most of the students mentioned that they would like to use the system 
for learning in future. 

We believe that a system like CAS will help a great deal in improving the con¬ 
tent of the textbooks by getting feedback from the interaction of the students. It 
will also influence the creation of textbooks in future by showing an integration 
of text, collaboration and opinion. The usefulness of the interface will be even 
beyond the knowledge refinement. It will be fit for use in many other domains 
as well, where any type of collaboration or decision making based on the exist¬ 
ing facts is required. For example, the system will be usable for the review of 
research papers created by various researchers, for collaboration of doctors on 
a complex medical case etc. The platform will also help as a perfect infrastruc¬ 
ture for the reviewers of any conference/journal to discuss their views about a 
particular paper. 
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